In this chapter, we provide the foundations of the QIIME 2 approach to bioinformatic and statistical analysis of microbiome data. We first provide an overview of QIIME 2 (Sect. 1.1). Then we introduce the core concepts in QIIME 2 (Sect. 1.2). Next, we introduce how to install QIIME 2 (Sect. 1.3). Following that, we introduce how to store and track data in QIIME 2 (Sect. 1.4) and extract data from QIIME 2 (Sect. 1.5). We complete this chapter by a brief summary (Sect. 1.6).
1.1 Overview of QIIME 2
QIIME (canonically pronounced chime: Quantitative Insights Into Microbial Ecology) is an open source microbiome bioinformatics platform designed for analyzing microbial ecological communities (J. G. Caporaso et al. 2010). It has been used in many microbiome studies including analysis of bacterial, archaeal, fungal, or viral sequence data. QIIME analysis generally starts with raw sequence data (in FASTA format) generated from any sequencing technology. QIIME scripts primarily wrap other software packages. It can analyze high-throughput data in a wide variety of ways. QIIME is implemented as a collection of command-line scripts designed to take users from raw sequence data and sample metadata through publication-quality graphics and statistics (Xia et al. 2018). QIIME 2 has succeeded QIIME 1 on January 1, 2018. QIIME 1 is no longer supported since end of 2017. QIIME 2 (Bolyen et al. 2019) is not only a complete redesign and rewritten version of the QIIME 1, but also a completely reengineered and rewritten system aiming to facilitate reproducible and modular analysis of microbiome data.
- (1)
QIIME 2 is a bioinformatic analysis tool via wrapping other sequencing platforms, such as DADA2 (Callahan et al. 2016) and Deblur (Amir et al. 2017) to perform analysis, data generation, sequence quality control, taxonomy assignment, phylogenetic insertion, and other functions. This topic is one core part of this book, which is substantially introduced in Chaps. 3, 4, 5, and 6.
- (2)
QIIME 2 is also able to perform basical statistical analysis of microbiome data, such as calculating and conducting alpha diversity and beta diversity analysis. This part of contents is introduced in Chaps. 9, 10, 11, 14, and 15, respectively.
- (3)
QIIME 2 can also be used to manage and visualize microbiome data and meta data, such as Sect. 3.5, and do emperor plots (see Sect. 10.4.2).
The QIIME 2 system architecture consists of three core components: the framework, the interfaces, and the plugins. Interfaces (q2cli, q2studio, and Artifact API) define how users interact with the system; plugins define all domain-specific functionality (such as q2-data2, q2-feature-table, and q2-types). Interfaces and plugins do not communicate directly with one another; it is the framework that mediates communication between plugins and interfaces and performs core functionality such as provenance tracking. For the details of how QIIME 2 system works, the interested reader can check the QIIME 2 website and Bolyen et al. (Bolyen et al. 2019).
QIIME is a python interface that combines many independent scripts for the analysis of microbiome data, allowing data to be analyzed all the way from raw sequence data to diversity indices and taxonomic breakdowns. QIIME 2 has some distinctive features that make it to be powerful and distinguish it from other open source software tools for microbiome data science.
First, QIIME 2 uniquely and powerfully wraps the data generation tools into plugins (software packages). Continuing the development of QIIME 1, QIIME 2 is developed based on a plugin architecture and wrapping other bioinformatics tools, providing its capabilities to analyze high-throughput data through various ways, such as barcode splitting, cleaning and filtering low-quality sequences, removing chimera sequences via USEARCH 6.1 (Edgar 2010), PyNAST alignment (J. Gregory Caporaso et al. 2009), taxonomic analysis, tree building, and clustering samples. Currently QIIME 2 has wrapped DADA2 and Deblur (the latest generation tools) for sequence quality control, taxonomy assignment, and phylogenetic insertion. Among a number of bioinformatics tools, QIIME and mothur (Schloss et al. 2009) were reviewed as the two outstanding pipelines (Nilakanta et al. 2014) because of their comprehensive features and support documentation. QIIME 2 is getting the most popular since its release of reengineered and rewritten system and currently has become the dominant tool for 16S microbiome data analysis.
Second, QIIME 2 provides basic statistical analysis, including alpha- and beta-diversify analyses, as well as supports qualitatively new functionality, such as microbiome paired sample, timeseries analysis, machine learning, compositional analysis, and gneiss analysis.
Third, QIIME 2 not only provides “upstream” processing steps (e.g., sequence demultiplexing and quality control) for generating data, but also provides many interactive visualization tools to facilitate exploratory analyses and provides publication-quality graphics for result reporting. The users can also through QIIME 2 View (https://view.qiime2.org) securely share and interact with results without installing QIIME 2.
Finally, QIIME 2 not only is a powerful marker gene analysis tool but also has the potential to serve as a multidimensional and powerful data science platform for multi-omic microbiome studies, for example, through equipping newly released plugins including q2-cscs, q2-metabolomics, q2-shogun, q2-metaphlan2, and q2-picrust2, and integrating other data types, such as metabolite or metatranscriptome profiles, allowing users to run MetaPhlAn2 through QIIME. QIIME 2 can perform initial analysis of metabolomics and shotgun metagenomics data. Thus, QIIME 2 can be rapidly adapted to analyze diverse microbiome features and multi-omics data.
1.2 Core Concepts in QIIME 2
In QIIME 2, an Action (i.e., methods, visualizers, and pipelines) creates a Result, and a Result can be either an Artifact or a Visualization. Each execution of an Action and all Results created by Actions are assigned with the version 4 universally unique identifiers (UUIDs). There are five core concepts in QIIME 2. We describe them below.
1.2.1 Artifacts
In QIIME 2, artifacts are defined as data files. The term artifact is an object that is made in a similar way as in an archaeological artifact. An Artifact is data generated by one or more Actions; or formally artifacts are the way of existence for data produced by QIIME 2. An artifact contains data and metadata. A metadata is the dataset that is used to describe data type, format, and provenance (i.e., how it was generated). QIIME 2 typically uses the .qza file extension to store an artifact in a file. QIIME 2 does not work with simple data files (e.g., FASTA files), instead works with artifacts. There are two benefits for using artifacts instead of simple data files: (1) automatically tracks the type, format, and provenance of data and (2) enables researchers to focus on conducting the analyses, instead of paying much attention to the particular data format. Because QIIME 2 directly works with artifacts, thus, when we import data, we must first create a QIIME 2 artifact. Typically we start importing raw sequence data, although data can be imported in any step of analysis. In QIIME 2, data also can be exported from an artifact.
1.2.2 Visualizations
Visualizations are another type of data generated by QIIME 2. QIIME 2 typically uses the .qzv file extension to store visualization files. Visualizations contain similar types of metadata as artifacts, including provenance information. Because in QIIME 2 both artifacts and visualizations include metadata and particularly have unique provenance information, so they can be archived or shared with collaborators. Both artifacts and visualizations files (generally with .qza and .qzv extensions, respectively) can be easily reviewed using the website (https://view.qiime2.org) without requiring a QIIME installation. The examples of visualizations are a statistical results table, an interactive visualization, static images, or really any combination of visual data representations. All these are terminal outputs of an analysis, which is contrast to artifacts. Thus, we cannot use visualizations as input to other analyses in QIIME 2.
1.2.3 Semantic Type
In QIIME 2, all Artifacts are annotated with a semantic description of their type. Data types present how data is represented in memory, and file formats present how data is stored on disk, whereas semantic types differ from data type and file formats, conveying the meaning of the data. The semantic types are used to (1) identify artifacts for suitable inputs to an analysis, which can prevent incompatible artifacts from being used in the analysis. It effectively constrains multiple actions to only those which are semantically meaningful actions. For example, in QIIME 2, phylogenetic trees have two semantic types: Phylogeny[Rooted] and Phylogeny[Unrooted]. The beta-phylogenetic action needs a UniFrac distance matrix, which works only on Phylogeny[Rooted], while fasttree can only generate a Phylogeny[Unrooted]. Thus the semantic type system is used to determine that the output of fasttree should not be directly provided as input to beta-phylogenetic. (2) They help users avoid incorrectly using data for semantically incorrect analyses. For example, categorical data cannot be used for a quantitative analysis, vice versa. (3) Also they facilitate users in identifying relevant workflows to generate desired data or further explore data.
1.2.4 Plugins
QIIME 2 implements microbiome analyses via plugins (software packages). Thus, at least one plugin needs to be installed to provide the specific analyses, such as the q2-demux plugin for demultiplexing raw sequence data, the q2-diversity plugin for alpha- or beta-diversity analyses. The initial plugins for an end-to-end microbiome analysis pipeline were developed by the QIIME 2 team; other plugins for additional analyses are developed by third-party developers and volunteers.
To perform microbiome analyses, QIIME 2 plugins must define methods and visualizers. The question is: how to define them? A method is defined to combine artifacts and parameters as input, which produces one or more artifacts as output. The resulted output artifacts could subsequently be used as input to other methods or visualizers. The outputs produced by methods can be intermediate or terminal. One example of method definition is as below: in the q2-feature-table plugin, the rarefy method is defined to use a feature table artifact and sampling depth as input, producing a rarefied feature table artifact as output. Then we could use the resulted rarefied feature table artifact to calculate alpha diversity by providing an alpha method in the q2-diversity plugin. A visualizer is similar to a method in the sense that it combines artifacts and parameters as input. However, a visualizer is distinctive to a method: it produces exactly one terminal visualization as output, i.e., the resulted visualizations cannot be used as input to other methods or visualizers.
1.3 Install QIIME 2
Currently QIIME 2 is available for macOS, Windows, and Linux users. All have three approaches of installation: a native conda installation, and Docker and VirtualBox. The native conda installation usually works well. Thus QIIME 2 generally recommends the method of native conda installation.
We can install QIIME 2 natively or use virtual machines. Below we introduce natively installing QIIME 2 (Core 2022.2 distribution). The readers can easily follow the instruction of QIIME 2 document to install QIIME 2 using virtual machines. The QIIME 2 Core 2022.2 distribution includes the QIIME 2 framework, q2cli (a QIIME 2 command-line interface), and the following plugins: q2-alignment, q2-composition, q2-cutadapt, q2-dada2, q2-deblur, q2-demux, q2-diversity, q2-diversity-lib, q2-emperor, q2-feature-classifier, q2-feature-table, q2-fragment-insertion, q2-gneiss, q2-longitudinal, q2-metadata, q2-phylogeny, q2-quality-control, q2-quality-filter, q2-sample-classifier, q2-taxa, q2-types, and q2-vsearch.
The recommended way to natively install QIIME 2 is through installing Miniconda. Miniconda provides the conda environment and package manager. QIIME 2 works within a conda environment. To install Miniconda in Mac, the following steps are needed:
Follow the Miniconda instructions for downloading and installing Miniconda. QIIME 2 works with either Miniconda2 or Miniconda3 (i.e., Miniconda Python 2 or 3). You may choose either one. Please follow the directions provided in the Miniconda instructions; particularly ensure that you run conda init and your Miniconda installation is fully installed.
Step 1: Click the website link https://docs.conda.io/en/latest/miniconda.html to download Miniconda (64-bit bash installer).
Step 2: Open a Mac Terminal (Terminal is within Utilities folder nested in Applications folder). Type: cd Downloads
Step 3: Run the bash “shell” script to install Miniconda. In the terminal, type: bash Miniconda3-latest-MacOSX-x86_64.sh
Step 4: Close the Terminal, and open a new Terminal. Type: conda -V
conda update conda
conda install wget
Step 5: Install QIIME 2 within the conda environment.
Since we have Miniconda installed, a conda environment is created. Now we are ready for installing QIIME 2 within this conda environment. We can natively install the QIIME 2 Core 2022.2 distribution within the environment. Because there are many required dependencies that are not needed to be added to an existing environment, QIIME 2 highly recommends creating a new environment specifically for the QIIME 2 release being installed. Here, we name the environment qiime2-2022.2 to indicate this is QIIME 2 Core 2022.2 distribution. We choose macOS/OS X (64-bit) and in the Terminal, type:
Step 6: Activate the conda environment.
To activate the QIIME 2 environment, type the environment’s name (here, qiime2-2022.2) in the Terminal:
Step 7: Test the installation.
To test the installation, activate the QIIME 2 environment and run:
If the help information comes out when running this command, suggesting that the installation was successful!
1.4 Store and Track Data
In QIIME 2, data are stored in a directory structure called an Archive with a single root directory (UUID) serving as the identity of the archive. Several sub-directories exist under the a single root directory (UUID), including (1) a data directory, (2) a metadata directory, (3) a provenance directory, and (4) a VERSION directory. The data directory contains only the data in a relevant format. For example, the fasta or fastq for sequence data, and newick for phylogenetic trees are stored in this data directory. All files in these archives are zipped for data delivery convenience and facilitating data sharing. For example, the extension .qza is for QIIME zipped artifact and the extension .qzv is for QIIME zipped visualization. We can use unzip, WinZip, 7Zip, or other common tools to unzip these standard zip files. The zipped files are easily shared by peers or collaborators, or submitted to journals.
QIIME 2 uses data provenance tracking to store all information about the series of Actions that led to a Result, packages, and Python dependencies and other environment information and the data itself. Microbiome data analyses usually generate many output files. Data-provenance tracking is used to avoid losing track of how each file was generated.
1.5 Extract Data from QIIME 2 Archives
QIIME 2 .qza and .qzv files are zip file containers with a defined internal directory structure. Depending on whether or not QIIME 2 and the q2cli command line interface are installed in your computer, you can use the qiime tools export command or use the standard decompression utilities such as unzip, WinZip, or 7zip to access these files. Any actual data files in the .qza artifact can be extracted directly using qiime tools export, which is basically just a wrapper for unzip. Alternatively, we can unzip the artifact directly using unzip -k file.qza from the files in the data folder.
1.6 Summary
In this introductory chapter of QIIME 2, we first provided an overview of QIIME 2. Then we described some core concepts in QIIME 2, including artifacts, visualizations, semantic type, and plugins. Next, we introduced how to install QIIME 2 step-by-step. Following that, storing, tracking data, and extracting data from QIIME 2 archives were introduced.
In Chap. 3 of this book, we will introduce some basic data processing procedures in QIIME 2. Chapter 4 will introduce building feature table and feature representative sequences from raw reads. Chapter 5 will introduce assigning taxonomy and building phylogenetic tree. Chapter 6 will introduce clustering sequences into OTUs. Before we move on to these chapters, let’s introduce some general R functions, packages, and specifically designed R packages for microbiome data analysis that are used in this book (Chap. 2).